Maitri Demonstration: Managing Large Scale Scientific Data (Demo)

Even traditional commercial database systems do not scale to the size of today’s large scientific data sets, whose growth is outpacing Moore’s Law. Instead, scientists are wedded to special-purpose data formats and their associated I/O libraries, even though these libraries provide only basic functionality. Thus there is a need for a scalable data management system that can support these formats and, when needed, provide more sophisticated functionality for indexing, buffering, caching, concurrency control, metadata management, and querying. This demonstration showcases Maitri, a framework that can be used to address these needs. The Maitri framework consists of a set of standard, very narrow interfaces for format-agnostic, loosely-coupled libraries offering aspects of the functionality listed above. Format independence is provided by Maitri’s block manager module, which encapsulates all code that is specific to a particular scientific data format, and calls the appropriate scientific I/O library to read and write data in that format. The demonstration shows the Rocketeer visualization toolkit running with subsets of the Maitri module implementations to visualize rocket simulation output data. The Rocketeer runs show the efficiency that can be gained by layering format-agnostic query, buffering, and/or indexing facilities atop scientific I/O libraries. The demonstration also shows that despite their narrow interfaces, the implementations of Maitri modules work together very effectively.